Piecewise Linear Multilayer Perceptrons and Dropout

We propose a new type of hidden layer for a multilayer perceptron, and demonstrate that it obtains the best reported performance for an MLP on the MNIST dataset. 1 The piecewise linear activation function We propose to use a specific kind of piecewise linear function as the activation function for a multilayer perceptron. Specifically, suppose that the layer receives as input a vector x ∈ R. The layer then computes presynaptic output z = xW + b where W ∈ R and b ∈ R are learnable parameters of the layer. We propose to have each layer produce output via the activation function h(z)i = maxj∈Sizj where Si is a different non-empty set of indices into z for each i. This function provides several benefits: • It is similar to the rectified linear units (Glorot et al., 2011) which have already proven useful for many classification tasks. • Unlike rectifier units, every unit is guaranteed to have some of its parameters receive some training signal at each update step. This is because the inputs zj are only compared to each other, and not to 0., so one is always guaranteed to be the maximal element through which the gradient flows. In the case of rectified linear units, there is only a single element zj and it is compared against 0. In the case when 0 > zj, zj receives no update signal. Preliminary work. • Max pooling over groups of units allows the features of the network to easily become invariant to some aspects of their input. For example, if a unit hi pools (takes the max) over z1, z2, and z3, and z1, z2 and z3 respond to the same object in three different positions, then hi is invariant to these changes in the objects position. A layer consisting only of rectifier units can’t take the max over features like this; it can only take their average. • Max pooling can reduce the total number of parameters in the network. If we pool with nonoverlapping receptive fields of size k, then h has size N/k, and the next layer has its number of weight parameters reduced by a factor of k relative to if we did not use max pooling. This makes the network cheaper to train and evaluate but also more statistically efficient. • This kind of piecewise linear function can be seen as letting each unit hi learn its own activation function. Given large enough sets Si, hi can implement increasing complex convex functions of its input. This includes functions that are already used in other MLPS, such as the rectified linear function and absolute value rectification.